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Executive Summary 


The National Center on Educational Outcomes (NCEO) held a virtual meeting of an Interim As- 
sessment Advisory Panel in February, 2021 to identify issues and make recommendations for state 
departments of education about how to enable all students with disabilities, including students with 
the most significant cognitive disabilities and English learners with disabilities, to participate in 
interim assessments in ways that yield valid inferences about what they know and can do. Advisors 
noted concerns and gaps about current interim assessments and their uses, suggested what should 
be, and proposed practical considerations for a better interim assessment system for each of five 
areas: (a) Participation; (b) Accessibility; (c) Role of academic standards; (d) Technical issues; 
and (e) Data use, interpretation, and reporting. 


Based on the advisors’ discussions, NCEO proposed seven recommendations for states and districts 
that use (or are considering use of) interim assessments. 


¢ Obtain data on the participation of students with disabilities in interim assessments, and 
develop a plan for analyzing those data. It is essential to examine participation rates and 
discuss these in ways that inform decision making. 


¢ Provide high-quality alternate interim assessments for students with the most significant 
cognitive disabilities. For alternate interim assessments to be worthy of administering, it is 
vital that they are high-quality assessments that measure what they are intended to measure, 
provide meaningful and useful data, and are engaging to the student. 


e Provide a wide range of accessibility features and accommodations, including accommoda- 
tions for students with sensory disabilities (e.g., braille, graphic organizers, sign language 
interpretation, etc.) and English learners with disabilities (e.g., glossaries, translations, 
etc.). Students with disabilities, including those with sensory disabilities and English learn- 
ers with disabilities, need to be able to access interim assessments in ways that allow them to 
meaningfully show what they know and can do. 


¢ Provide guidance on the provision of accessibility and accommodations. State education 
agencies (SEAs) and local education agencies (LEAs) need to engage with vendors to develop 
guidance and polices on accessibility and accommodations. 


e Provide transparency on the degree of alignment (or lack of alignment) to grade-level 
academic content standards of both general and alternate interim assessments. Documen- 
tation of the alignment of the assessment to grade-level academic content standards should be 
publicly provided. Requests for proposals (RFPs) and requests for information (RFIs) should 
require vendors to show how general and alternate interim assessments map onto a state’s 
grade-level academic content standards. 


Provide documentation and evidence that interim assessments are properly designed to 
evoke the intended cognitive processes when administered to students with disabilities. 
Documentation and validity evidence (e.g., think alouds, cognitive labs) should be provided 
to confirm that students with disabilities are demonstrating the assessments’ intended cogni- 
tive processes. 


Provide publicly available guidance on the appropriate and inappropriate uses of data 
in formats that meet the needs of intended audiences. Guides and reports that are created 
should consider the needs of intended audiences (e.g., district and school leaders, teachers, 
policymakers, parents, and families). Appropriate and inappropriate purposes and uses of data 
from each interim assessment should be publicly documented. 
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Overview 


The National Center on Educational Outcomes (NCEO) held a virtual meeting of an Interim 
Assessment Advisory Panel on February 16 and 17, 2021 to tap into the panel members’ col- 
lective knowledge about using interim assessments to support valid interpretations of what 
students with disabilities know and can do. The Interim Assessment Advisory Panel was com- 
posed of individuals with expertise in interim assessments and students with disabilities. They 
represented psychometricians, researchers knowledgeable about students with disabilities, state 
education agency (SEA) assessment and special education leaders, local education agency 
(LEA) leaders, and parents. See Appendix A for a list of the advisors, as well as a list of other 
meeting participants. 


In this report, interim assessments refer to assessments that are administered several times 
during a school year to measure student progress. They may be commercially produced or 
developed by groups of states or other organizations. Other terms that are sometimes used to 
describe these assessments are local assessments, district assessments, and even formative as- 
sessments (although interim assessments rarely provide formative information). 


The meeting purpose was to develop and disseminate guidance to state departments of educa- 
tion on how to enable all students with disabilities, including students with the most significant 
cognitive disabilities and English learners with disabilities, to participate in interim assess- 
ments in ways that yield valid inferences of what they know and can do. A specific goal of the 
meeting was to have the advisors discuss the issues related to this purpose. NCEO believed 
that these discussions would lead to the identification of current gaps or issues, suggestions for 
what should be, practical considerations, and some recommendations for policies and practices. 
Takeaways from the meeting and recommendations created by NCEO could inform state de- 
partments of education in their efforts to support improved measurement of what students with 
disabilities know and can do. Guidance from states can, in turn, inform decisions of districts 
and Individualized Education Program (IEP) teams, and ultimately improve the measurement 
of student outcomes. 


Background Information 


Interim assessments are used for a variety of purposes. Some of these include: 
¢ To measure growth or progress 
¢ For instructional decision making 


* To predict summative assessment performance 
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¢ To measure whether a student meets a grade-level goal 


Additionally, widespread interest is expressed by states and districts about the possible use of 
interim assessments to measure learning losses that may have resulted because of COVID-19 
school closures and inconsistent distance learning. Some states also are interested in potentially 
using either commercially-produced or state-developed interim assessments as a replacement 
or supplement to state tests used for accountability during this and other times. However, little 
attention has been given to how to include some students with disabilities in these assessments 
in ways that produce valid results and support valid interpretations of those results. Further, 
there are federal requirements that all state- and district-wide administrations of an assessment 
must have an alternate assessment, so states that adopt interim assessments will need to imple- 
ment approaches that are appropriate for students who participate in their alternate assessments. 


Because there is so much interest in using interim assessments, there is an urgent need for 
guidance to support states and districts as they make decisions about the use of interim assess- 
ments to measure progress of students with disabilities as well as to inform instruction, predict 
performance, and measure learning loss. The Interim Assessment Advisory Panel discussions 
summarized in this report were to help NCEO develop recommendations for states. The recom- 
mendations will also benefit those states that already use some form of interim assessments to 
measure progress as part of the U.S. Department of Education’s Office of Special Education 
Programs’ Results-Driven Accountability (RDA) system. 


Process 


Fourteen advisors participated in the meeting. Additionally, six NCEO staff members, five NCEO 
partners, and three consultants attended the meeting. NCEO staff, partners, and consultants 
gave presentations that provided background information, facilitated advisor discussions, and 
provided meeting support. Seven participants from the U.S. Department of Education’s Office 
of Special Education Programs (OSEP) and Office of Elementary and Secondary Education 
(OESE) attended the meeting as well. 


To prepare for this meeting NCEO supported a scan of the interim assessment landscape. It 
focused on the publicly available documentation supporting the appropriateness of these assess- 
ments for students with disabilities (Boyer & Landl, 2021). It also compiled information about 
how interim assessments were being used by states for OSEP RDA accountability (Lazarus et al., 
2021). These sets of information were shared with advisory panel members prior to the meeting. 


The Interim Assessment Advisory Panel Meeting was held for five hours on each of two days 
(see meeting agenda in Appendix B). The first day of the meeting alternated between presenta- 
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tions and breakout discussions. (The Facilitator Guide for the breakout sessions is provided in 
Appendix C). Advisors considered the needs of three groups of students throughout the meeting: 
students with disabilities in general, students with the most significant cognitive disabilities, 
and English learners with disabilities. 


During the second day of the meeting, facilitators supported the advisors as they discussed the 
issues and created takeaways. Using the meeting takeaways, NCEO developed a set of recom- 
mendations for states. 


NCEO provided advisors with a framework to help organize their thinking about interim as- 
sessments for students with disabilities (including those with sensory disabilities), students 
with the most significant cognitive disabilities, and English learners with disabilities. The five 
components of the framework, and the general question for each component were: 


¢ Participation: How can we better understand current interim assessment participation 
practices? 


¢ Accessibility: How can we better understand current accessibility policies and decision- 
making processes? 


* Role of Academic Standards: What are the standards-based issues that need to be considered 
when using interim assessments to measure what students with disabilities know and can do? 


¢ Technical Issues: What are the technical issues that need to be considered when using interim 
assessments to measure what students with disabilities know and can do? 


¢ Interpretation and Use of Data: What needs to be considered for the interpretation and use 
of data for students with disabilities from interim assessment? 


After the meeting, NCEO summarized the advisors’ comments and then sent them back out to 
the advisors to review and suggest revisions. Based on this summary, NCEO produced a draft 
of this report, which advisors also reviewed before publication. 


Structure of this Report 


This report summarizes both the overview information provided to meeting participants and 
the advisors’ discussions in the form of takeaways. NCEO staff developed summaries of the 
presentations from notes taken during the presentations and from the presenters’ slides (see 
Appendix D for a summary of the proceedings). NCEO also developed summaries of the fa- 
cilitated discussions from notes taken by note takers. These are reflected in the next section of 
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this report. The advisors were encouraged by a facilitator to comment and discuss freely, with 
assurances from NCEO that the final report would not attribute any particular comment to any 
specific advisor. This led to frank and open conversations. 


Summary of Advisory Panel Members’ Discussion and Takeaways 


Interim Assessment Advisory Panel members expressed concerns about the quality of interim 
assessments and their use. They were concerned that many of the commercially available interim 
assessments had incomplete evidence to support the validity of their intended interpretations and 
uses. This led some advisors to question whether students with disabilities should be included 
in them, and whether an alternate assessment should be developed for students with the most 
significant cognitive disabilities. Despite these concerns, the discussion during the meeting fo- 
cused on the five topics in recognition of federal requirements for the inclusion of all students 
with disabilities in state and districtwide assessments, either in the general assessment, with or 
without accommodations, or in an alternate assessment. 


This section first presents the legal background that framed the advisors’ discussion about each 
of the five components of the framework. This is followed by a summary of the advisors’ dis- 
cussion. The discussion summary for each component is organized into three parts: (a) Current 
Concerns or Gaps; (b) What Should Be; and (c) Practical Considerations for a Better System. 


Participation 


Federal laws require that all children with disabilities be included in all state and districtwide 
administrations of interim assessments. This includes students who need appropriate accommo- 
dations to participate in the general assessment and students with the most significant cognitive 
disabilities who may need an alternate assessment, as well as English learners with disabilities. 
Regulations for the Individuals with Disabilities Education Act (IDEA) state: 


A State must ensure that all children with disabilities are included in all general State 
and district-wide assessment programs, including assessments described under section 
1111 of the ESEA, 20 U.S.C. 6311, with appropriate accommodations and alternate as- 
sessments, if necessary, as indicated in their respective IEPs. (Sec. 300.160(a)) 


If an interim assessment is used for federal accountability, the requirements of the Elementary 
and Secondary Education Act (ESEA), reauthorized in 2015 as the Every Student Succeeds Act 
(ESSA), also apply. ESSA requires the inclusion of all students in assessments used for account- 
ability (Sec. 1111(2)(B)()(ID). For English learners with disabilities, participation requirements 
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are reinforced by several civil rights laws and court cases (e.g., Title VI of the Civil Rights Act 
of 1964, Lau v. Nichols, 414 U.S. 563 (1974)). 


Advisors identified several current concerns or gaps, what should be, and practical consider- 
ations for a better system. 


Participation—Current Concerns or Gaps 


¢ Many students with disabilities may be excluded from interim assessments, especially stu- 
dents with sensory disabilities, due to insufficient accessibility features and accommodations. 


¢ There is a lack of public data on the participation of students with disabilities in interim 
assessments. 


¢ There are no known alternate interim assessments for students with the most significant 
cognitive disabilities. 


¢ If students with disabilities are excluded from interim assessments, the resulting data will 
not be representative of all students. 


¢ If some students with disabilities are excluded from interim assessments the data may be 
less useful for many purposes (e.g., measuring progress, instructional decision making, 
accountability, etc.). 


¢ Students with disabilities are more likely to be excluded from interim assessments, and thus 
may be disproportionately denied access to classes or courses that use such assessments to 
determine eligibility for enrollment. 


Participation—What Should Be 


¢ All students with disabilities should be able to participate in interim assessments with needed 
accessibility features and accommodations. 


¢ Alternate assessments for students with the most significant cognitive disabilities should be 
available for all state or districtwide assessments. 


Participation—Practical Considerations for a Better System 


¢ All state and districtwide administrations of interim assessment should include both a gen- 
eral interim assessment with appropriate accessibility features and accommodations, and 
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an alternate interim assessment for students with the most significant cognitive disabilities 
who are unable to access the general assessment even with accommodations. 


SEAs and LEAs should examine their current participation data (in terms of numbers and 
percentages) to identify the degree to which groups of students with disabilities (e.g., English 
learners) are excluded from interim assessments. Students who take the state’s alternate 
assessment based on alternate academic achievement standards (AA-AAAS) and who also 
participate in existing interim assessments should be identified. 


Because commercially-provided interim assessments are in widespread use, requests for 
proposals (RFPs) and requests for information (RFIs) should require interim assessment 
vendors to provide participation data to the public, including information about participa- 
tion by student subgroups (e.g., students with disabilities, by category of disability, and if 
possible, English learners with disabilities, students who take the assessment with accom- 
modations, etc.). 


SEAs and LEAs should ask vendors to provide an alternate assessment, or should develop 
one, for any state or districtwide assessment to meet IDEA requirements. 


SEAs and LEAs should work with vendors to merge state or district and vendor files in 
a way that will allow for ongoing documentation of participation in interim assessments. 


SEAs and LEAs should communicate with stakeholders (e.g., educators, parents) and pro- 
fessional organizations (e.g., AERA special interest group on inclusion and accessibility in 
educational assessments) about why it is important for students with disabilities to participate 
in interim assessments across multiple vendors and platforms. This may include providing 
evidence of the benefits to students of participating meaningfully in the assessments (e.g., 
for instructional decision making, measuring progress, access to services for which deci- 
sions are based on interim assessment, etc.). 


SEAs and LEAs should consider providing resources to educators to help support mean- 
ingful measurement practices for students with the most significant cognitive disabilities. 


Accessibility 


Federal law requires that states (or in the case of districtwide assessments, LEAs) develop ac- 


commodations guidelines that address the provision of appropriate accommodations for students 


with disabilities. According to IDEA regulations: 


(1) A State (or, in the case of a district-wide assessment, an LEA) must develop guide- 
lines for the provision of appropriate accommodations. 
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(2) The State’s (or, in the case of a district-wide assessment, the LEA’s) guidelines must— 


(i) Identify only those accommodations for each assessment that do not invalidate 
the score; and 


(11) Instruct IEP Teams to select, for each assessment, only those accommodations 
that do not invalidate the score. (Sec. 300.160(b)) 


If an interim assessment is used for federal accountability, ESSA requirements would also ap- 
ply. ESSA requires that states make appropriate accommodations available and ensure that their 
assessments are accessible to students with disabilities (Sec. 1111(2)(B)(vii)(ID). 


Advisors identified several current concerns or gaps, what should be, and practical consider- 
ations for a better system. 


Accessibility—Current Concerns or Gaps 


* Accessibility features and accommodations used by some students during instruction are 
not available for many interim assessments (e.g., braille, large print, text-to-speech, etc.). 


¢ Often there is a lack of clarity about interim assessment accessibility and accommodations 
policies. 


¢ Vendor approaches to accessibility and accommodations vary, which can lead to inconsistent 
availability of accessibility features across LEAs. 


¢ There is a lack of clarity about who is responsible for developing accessibility and accom- 
modations policies for interim assessments. For example, LEAs may have their own policies, 
which may or may not align with vendor recommendations or state policies. 


¢ From the perspective of LEAs and vendors, the costs associated with making assessments 
accessible and providing alternate forms (e.g., braille, large print, translations) may seem 
to outweigh the benefits. 


Accessibility—What Should Be 


¢ High-quality assessments should be accessible to all students, including (but not limited to) 
students with sensory disabilities, students with the most significant cognitive disabilities, 
and English learners with disabilities. 


¢ Universal design for assessment (UDA) principles should be used to develop interim as- 
sessments so that there is little need for retrofitting. 
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¢ Validity studies that investigate the appropriateness of accessibility features and accom- 
modations should be conducted and documented by interim assessment vendors. 


¢ Interim assessment accessibility features and accommodations should not differ significantly 
across LEAs. 


¢ Interim assessments should have a publicly available accessibility policy. 


¢ Educators and IEP teams should receive clear information about accessibility and accom- 
modation policies for interim assessments from SEAs, LEAs, and vendors so educators and 
IEP teams are not making decisions on the fly. 


Accessibility—Practical Considerations for a Better System 


¢ RFPs and RFIs should require interim assessment vendors to provide clear evidence that a 
wide range of accessibility features and accommodations are available. They should also 
require vendors to provide evidence that available accessibility features and accommodations 
support valid results and interpretations for the purposes for which the interim assessments 
will be used. These accessibility features and accommodations should be appropriate and 
effective for meeting individual student needs, and allow for meaningful and valid interpre- 
tations and uses of the results and scores. 


¢ RFPs and RFIs should require application of UDA principles in all stages of the assessment 
design and development processes. They should also require vendors to provide empirical 
evidence of how UDA principles have been applied. 


¢ RFPs and RFIs should require vendors to implement accessibility standards that are com- 
pliant with Section 508 of the federal Rehabilitation Act (e.g., Web Content Accessibility 
Guidelines [WCAG]) in their interim assessments. 


¢ SEAs, LEAs, and vendors should provide guidance and training for educators, including 
those in preservice programs, on selecting, implementing, and evaluating assessment ac- 
cessibility features. 


¢ SEAs and LEAs should provide guidance to IEP teams on making accessibility decisions 
for interim assessments. 


8 NCEO 


Role of Academic Standards 


Federal laws require that all children with disabilities have the opportunity to learn grade-level 
academic content. This includes students with the most significant cognitive disabilities who 
may need an alternate assessment, and English learners with disabilities. 


According to IDEA, all students with disabilities must have meaningful access to content aligned 
with the State’s academic content standards for the grade in which the child is enrolled.! ESSA 
also requires that students with disabilities participate in academic instruction and assessments 
for the grade level in which the student is enrolled, and are tested based on challenging State 
academic standards for the grade level in which the student is enrolled (Sec. 1111(2)(B)(ii)). 


Some interim assessments may not be aligned to a state’s grade-level academic content stan- 
dards. The purpose for which an assessment is being used affects whether it needs to be aligned 
to grade-level standards. If the intent is to determine whether students are learning grade-level 
academic content, the interim assessment should be aligned to the standards. 


Advisors identified several current concerns or gaps, what should be, and practical consider- 
ations for a better system. 


Role of Academic Standards—Current Concerns or Gaps 


¢ It cannot be assumed that SEA and LEA leaders understand the challenges involved in ad- 
equately aligning interim assessments to the academic content standards students are taught. 


¢ Data from interim assessments are sometimes used for purposes that go beyond the purposes 
for which the assessment was designed. 


* Correlation of some interim assessments with some constructs of importance is not the same 
as alignment to the standards. The failure to provide well-aligned measures diminishes the 
value of interim assessments. 


¢ Stakeholders need clarity on the role of academic content standards when an interim assess- 
ment is used for screening purposes. 


'https://www2.ed.gov/policy/speced/guid/idea/memosdcltrs/guidance-on-fape-11-17-2015. 
pdf 
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Role of Academic Standards—What Should Be 


SEAs and LEAs should clearly understand the purpose for which they will use an interim 
assessment, and vendors should provide evidence that the assessment selected is designed 
and technically adequate for that purpose. 


Vendors should be required to make the degree of alignment (or lack of alignment) to 
grade-level academic content standards transparent for both general and alternate interim 
assessments. Ideally, alignment evidence would come from an independent source following 
established standards for professional practice. 


Role of Academic Standards—Practical Considerations for a Better System 


Both general and alternate interim assessments should include measures that are aligned 
to grade-level academic content standards and coherent with curriculum and instruction. 


SEA and LEA staff who select assessments need to have a good understanding of how 
standards, instruction, and assessment all work together. 


RFPs and RFIs should require vendors to show how both general and alternate interim 
assessments map onto state grade-level content standards, as well as correlations between 
performance on the interim assessment and performance on state standards-based assess- 
ments, if the purpose of the interim assessment is prediction. 


SEAs and LEAs should consider providing resources (e.g., trainings and support) to help 
improve the link between teaching, learning, and assessment. 


Technical Issues 


The U.S. Department of Education’s Office of Elementary and Secondary Education (OESE) 
conducts peer reviews of states’ assessments used for accountability. According to A States 


Guide to the U.S. Department of Education's Assessment Peer Review Process (U.S. Depart- 


ment of Education, 2018): 
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The purpose of the Department’s peer review of State assessment systems is to support 
States in meeting statutory and regulatory requirements under Title I of the Elementary 
and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Suc- 
ceeds Act (ESSA), for implementing valid and reliable State assessment systems. Under 
sections 1111(a)(4) and 1111(b)(2)(B)(i11)-(iv) of the ESEA and 34 CFR § 200.2(b) 
(4) and (5) and (d), the Department has an obligation to conduct a peer review of the 
technical quality of State assessment systems implemented under section 1111(b)(2) of 
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the ESEA. Assessment peer review is the process through which a State demonstrates 
the technical soundness of its assessment system. A State’s success with its assessment 
peer review begins and hinges on the steps the State takes to develop and implement a 
technically sound State assessment system. (p. 4) 


The federal peer review process is used for state summative assessments used for accountability. 
The peer review process results in assessments that are technically stronger. Historically there 
has sometimes been less emphasis on the technical soundness of interim assessments, but as 
they increase in profile and are used for new purposes, there is recognition that it is important to 
consider technical issues for these assessments. Frequently cited evidence of technical adequacy 
includes item statistics, reliability and measurement error, differential item functioning, factor 
analysis, linking and equating, and correlation studies. Particular attention needs to be given to 
validity evidence for interim assessments. 


Test scores should have the same meaning for a student with disabilities as they do for other 
students. In order to draw valid inferences from an assessment, it is important that an assessment 
produce valid interpretations for the purpose of the assessment. The fairness and equity of an 
assessment, the comparability of different forms (including alternate formats such as braille), the 
appropriateness and effect of various accessibility features and accommodations, and other issues 
related to the validity of score interpretations and uses all merit evaluation and documentation. 


Advisors identified several current concerns or gaps, what should be, and practical consider- 
ations for a better system. 


Technical Issues—Current Concerns or Gaps 


¢ There are sometimes gaps between the claims vendors make about the validity of using 
their interim assessments for measuring what students with disabilities know and can do, 
and existing documentation. 


¢ Students with disabilities often are not included in studies that are conducted to provide 
information about the technical characteristics of an assessment, including the validity of 
score interpretations and uses. 


¢ Interim assessment data are sometimes misused to attribute gains in performance to mastery 
of the standards-based content from year to year. 
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Technical Issues—What Should Be 


The needs of students with disabilities should be considered at the beginning and throughout 
the assessment design and development process, improving the extent to which all students 
can access the assessment tool. 


Validity studies that include students with disabilities should be conducted. 


Vendors should be required to show that their assessments are properly designed to evoke 
the intended cognitive processes when administered with the use of the appropriate acces- 
sibility features or accommodations. 


The assessment development process (e.g., item development, pilots, field testing, reviews, 
etc.) should be made more transparent and presented in appropriate formats so educators 
know and can communicate the trustworthiness of the assessment, including how acces- 
sibility features were addressed during the development process. Validity evidence for the 
accessibility features should be documented and made publicly available. 


Technical Issues—Practical Considerations for a Better System 
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Interim assessment vendors should use the Standards for Educational and Psychological 


Testing (AERA/APA/NCME, 2014) when developing assessments and providing evidence 
of technical adequacy and validity. 


The needs of students with disabilities, including (but not limited to) students with sensory 
disabilities, students with the most significant cognitive disabilities, and English learners with 
disabilities, should be considered throughout the interim assessment development process. 


SEA and LEA RFPs and RFIs should require vendors to provide detailed information about 
the technical elements of the interim assessments they are considering purchasing, including 
evidence (e.g., via the use of think alouds or cognitive labs) that the assessments are eliciting 
performance on the intended constructs from students with disabilities, including (but not 
limited to) students with sensory disabilities, students with the most significant cognitive 
disabilities, and English learners with disabilities. Detailed information on the technical and 
validity elements of alternate interim assessments should also be required. 


Public confidence in interim assessments can be improved with transparency. Knowledge of 
the assessment development processes would allow SEAs and LEAs to communicate with 
stakeholders about the validity of assessment results, interpretations, and uses, the reliability 
of scores, and the assessment and the accessibility features and accommodations options 
available for groups of students with disabilities. 
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Data Use, Interpretation, and Reporting 


Data from interim assessments are used for many purposes, and have both intended and unin- 
tended consequences. When using assessment data, considerations include context, data quality, 
intended purpose, data limitations, and consequences. If there is public reporting of interim 
assessment data, data also need to be reported for students with disabilities. 


IDEA requires that if a state or a district publicly reports assessment data for students without 
disabilities, then it must report assessment data for students with disabilities to the public with 
the same frequency and in the same detail. This includes reporting on participation and per- 
formance in regular and alternate assessments, as well as the number of students participating 
with accommodations. A state is eligible for IDEA funding only if it provides assurances to the 
Department of Education that it takes the following steps: 


(D) REPORTS.—The State educational agency (or, in the case of a districtwide assess- 
ment, the local educational agency) makes available to the public, and reports to the 
public with the same frequency and in the same detail as it reports on the assessment of 
nondisabled children the following: 


(i) The number of children with disabilities participating in regular assessments, and the 
number of those children who were provided accommodations in order to participate 
in those assessments. 


(ii) The number of children with disabilities participating in alternate assessments de- 
scribed in subparagraph (C)(ii)(I). 


(iii) The number of children with disabilities participating in alternate assessments de- 
scribed in subparagraph (C)(ii)(ID). 


(iv) The performance of children with disabilities on regular assessments and on alternate 
assessments (if the number of children with disabilities participating in those assessments 
is sufficient to yield statistically reliable information and reporting that information will 
not reveal personally identifiable information about an individual student), compared 
with the achievement of all children, including children with disabilities, on those as- 
sessments. (20 U.S.C. § 1412(a)(16)) 


Advisors identified several current concerns or gaps, what should be, and practical consider- 
ations for a better system. 
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Data Use, Interpretation, and Reporting—Current Concerns or Gaps 


Data from interim assessments are sometimes used for purposes for which they were not 
intended to be used. 


Stakeholders (e.g., LEAs, school-based staff, parents, IEP teams) are often unaware of data 
limitations and contextual factors when using interim assessment data for decision-making 
purposes. 


Although some interim assessments may be accurate at predicting performance on sum- 
mative assessments for most students, those with disabilities are often excluded from the 
assessment development process. These students are often excluded from participating in the 
assessments due to lack of accessibility. Also, some students with disabilities are included in 
interim assessments, but due to lack of appropriate accessibility features and accommoda- 
tions or alternate assessment, their test results are not a valid measure of what they know 
and can do and are not accurate at predicting performance on summative assessments. 


SEA staff often are unable to provide accurate guidance to stakeholders about the use of 
interim assessment data because of the wide variety of interim assessments from which 
LEAs can choose. 


Interim assessment data often are used to draw conclusions across multiple interim assess- 
ments without attention to differences in scales, number of achievement levels, and the labels 
used for those achievement levels. 


Vendors and LEAs sometimes link their interim data to statewide summative data even 
though this may not be appropriate. 


Data Use, Interpretation, and Reporting—What Should Be 
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Vendors must be clear about and publicly document the limitations of interim assessment 
data, as well as the appropriate purposes and uses of data from each assessment. They must 
publicly document inappropriate uses and discourage the use of data in unintended or inap- 
propriate ways. 


SEAs, LEAs, and vendors should routinely include information about data limitations and 
contextual factors in reports. 


SEAs and LEAs should adopt policies that require the use of multiple pieces of evidence 
when making high-stakes decisions. 
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Data Use, Interpretation, and Reporting—Practical Considerations for a Better System 


¢ Test users (e.g., SEAs, LEAs) and vendors should engage in purposeful coordination and 
collaboration to ensure appropriate data use, interpretation, and reporting. 


¢ SEAs, LEAs, and vendors can build confidence in interim assessment score reports by includ- 
ing information about both the appropriate and inappropriate interpretations and uses of the 
reported data as well as data limitations, and by showing they used a thoughtful, deliberate 
process for developing the reports. 


¢ Vendors should provide guides to understanding assessment results to educators, families, 
and other stakeholders. Interpretation guides and other materials should use language ap- 
propriate for the intended audience. 


¢ SEAs should post resources on their website that show how interim assessments fit into the 
larger assessment system. 


Recommendations 


Based on takeaways from the Interim Assessment Advisory Panel discussions, NCEO proposes 
the following recommendations for states and districts that use (or are considering use of) in- 
terim assessments. 


¢ Obtain data on the participation of students with disabilities in interim assessments, 
and develop a plan for analyzing those data. Currently, limited data are publicly avail- 
able on the participation of students in interim assessments. It is essential to examine par- 
ticipation rates and discuss these in ways that inform decision making. States and districts 
should learn more about the participation of students with disabilities in interim assess- 
ments. Questions to ask include: Are some students with disabilities excluded from interim 
assessments (e.g., students with IEPs; students in separate settings, etc.)? Are students in 
some disability categories less likely to be included in interim assessments (e.g., students 
with visual impairments or blindness, students who are deaf or hard of hearing, etc.)? Are 
results used appropriately for students included in interim assessments? 


e Provide high-quality alternate interim assessments for students with the most signifi- 
cant cognitive disabilities. Federal laws require that there be alternate assessments for all 
state and districtwide interim assessments. For these alternate assessments to be worthy of 
administering, it is vital that they are high-quality assessments that measure what they are 
intended to measure, provide meaningful and useful data, and are engaging to the student. See 
Alternate Interim Assessments for Students with the Most Significant Cognitive Disabilities 


NCEO 15 


16 


(Browder et al., 2021) for additional information about alternate interim assessments, and 
processes that could be used to develop them. 


Provide a wide range of accessibility features and accommodations, including accom- 
modations for students with sensory disabilities (e.g., braille, graphic organizers, sign 
language interpretation, etc.) and English learners with disabilities (e.g., glossaries, 
translations, etc.). Students with disabilities, including those with sensory disabilities and 
English learners with disabilities, need to be able to access interim assessments in ways that 
allow them to meaningfully show what they know and can do. 


Provide guidance on the provision of accessibility and accommodations. SEAs and 
LEAs need to engage with vendors to develop guidance and policies on accessibility and 
accommodations. 


Provide transparency on the degree of alignment (or lack of alignment) to grade- 
level academic content standards of both general and alternate interim assessments. 
Documentation of the alignment of the assessment to grade-level content standards should 
be publicly provided. This will assist potential users in making decisions about whether an 
assessment is appropriate for a given purpose, as well as increase public confidence. RFPs 
and RFIs should require vendors to show how general and alternate interim assessments 
map onto state grade-level content standards, as well as correlations between performance 
on the interim assessment and performance on state standards-based assessments if they 
are used for predictive purposes. 


Provide documentation and evidence that interim assessments are properly designed to 
evoke the intended cognitive processes when administered to students with disabilities. 
Documentation and validity evidence (e.g., think alouds, cognitive labs) should be provided 
as evidence that students with disabilities, including (but not limited to) students with sensory 
disabilities, students with the most significant cognitive disabilities, and English learners 
with disabilities, are demonstrating the assessments’ intended cognitive processes. 


Provide publicly available guidance on the appropriate and inappropriate uses of data in 
formats that meet the needs of intended audiences. The limitations of interim assessment 
data should be publicly documented, as well as the appropriate and inappropriate purposes 
and uses of data from each assessment. This information should discourage the use of data 
in unintended or inappropriate ways. Guides and reports that are created should consider 
the needs of intended audiences (e.g., district and school leaders, teachers, policymakers, 
parents, and families). 


NCEO 


Conclusions 


There is widespread interest in using interim assessments for many purposes, including using 
them for instructional decision making, to monitor progress, to measure learning loss, and to 
predict success on a summative test. These assessments are currently used as part of OSEP’s 
RDA accountability system. Many State Systemic Improvement Plans (SSIPs) include interim 
assessments either as the State Identified Measurable Result (SIMR) or as a measure of progress 
in the evaluation plan. Some states are also interested in using interim assessments as a replace- 
ment or supplement to summative tests used for ESSA accountability. 


Federal laws require that all state and districtwide interim assessment administrations include 
an alternate assessment, yet there currently are no alternate interim assessments. Federal laws 
also require that accommodations be provided for students with disabilities who need them, 
but many interim assessments do not offer a full range of accommodations, especially those 
that would enable students with sensory impairments to participate in the assessment. There 
is often a lack of transparency about whether an interim assessment is aligned to grade-level 
academic content standards, as well as whether these assessments are properly designed to elicit 
the intended constructs and cognitive processes when administered to students with disabilities. 


More work is needed to ensure that interim assessments provide valid information on what 
students with disabilities know and can do. Because so many states and districts use interim as- 
sessments—and more are considering using them—there is an urgent need for validity evidence 
for each of their uses and additional guidance on how states can effectively use and implement 
these assessments and provide training for districts and schools. 
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Meeting Agenda 


Interim Assessment Advisory Panel Meeting 


Agenda 


February 16-17, 2021 


Tuesday, February 16 


Welcome and Opening Remarks 
11:00-noon Sheryl Lazarus (Director, NCEO) 
David Egnor (Project Officer, OSEP) 
Christine Pilgrim (MSIP, OSEP) 
Donald Peasley (OESE) 
Everyone 


noon — 12:15 Meeting Overview 
Sheryl Lazarus (NCEO) 

12:15- 12:30 Meeting Processes and Specific Areas that Need to be Considered to Ensure that In- 
terim Assessments Yield Valid Inferences About What Students with Disabilities Know 
and Can Do 

Participation 


Accessibility 
Role of standards 


Technical Issues (e.g., reliability, validity, fairness, comparability, aggregation 
across multiple interim assessments, etc.) 
Interpretation and use of data 


Rachel Quenemoen 
Sheryl Lazarus 


12:30- 1240 


How Students with Disabilities are Included in Interim Assessments 
12:40 — 1:00 
Erika Landl and Michelle Boyer (NCIEA) 


Rachel Quenemoen 
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OSEP Results Driven Accountability, State Systemic Improvement Plans (SSIPs), and 
Assessment-related State Identified Measurable Results 
Susan Hayes, Kate Nagle, and Sheryl Lazarus 


2:50-3:50 Advisor Sharing (Breakout Groups) 
3:50-4:00 Summary of Day 
Sheryl Lazarus 


Wednesday, February 17 


Reflections on Day 1 

11:00-11:05 Sheryl! Lazarus 

11:10-11:15 Process for Developing Recommendations, and Introduction to Notes 
Rachel Quenemoen 

11:15-noon Participation 


noon — 12:45 


e Review of Notes 

e Suggestions for Recommendations 
o ESSA accountability and use by states and districts 
o IDEA SSIPs/SIMRs 


Facilitated by Rachel Quenemoen 


Accessibility 
e Review of Notes 
e Suggestions for Recommendations 
o ESSA accountability and use by states and districts 
o IDEA SSIPs/SIMRs 


Facilitated by Rachel Quenemoen 


TA8-12:58 


12:55-1:45 


22 


Role of Standards 
e Review of Notes 
e Suggestions for Recommendations 
o ESSA accountability and use by states and districts 
o IDEA SSIPs/SIMRs 


Facilitated by Rachel Quenemoen 
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1:45-2:30 Technical Issues (e.g., Reliability, Validity, Fairness, Comparability, Aggregation Across 
Multiple Interim Assessments, Etc.) 

e Review of Notes 

e Suggestions for Recommendations 


o ESSA accountability and use by states and districts 
o IDEA SSIPs/SIMRs 


Facilitated by Rachel Quenemoen 
2:40 —3:25 Interpretation and Use of Data 
e Review of Notes 
e Suggestions for Recommendations 


o ESSA accountability and use by states and districts 
o IDEA SSIPs/SIMRs 


Facilitated by Rachel Quenemoen 
Other Suggestions for Recommendation 
e Review of Notes 
e Suggestions for Recommendations 


o ESSA accountability and use by states and districts 
o IDEA SSIPs/SIMRs 


Facilitated by Rachel Quenemoen 


Next Steps 
3:40-3:50 Sheryl Lazarus 
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Breakout Group Facilitator Guide 


February 16, 2021 
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Facilitation Notes for the Breakout Group Sessions 


Breakout Session Process Set-up 

1:00 - 1:10: Rachel Quenemoen shares process for breakout discussion. 
Rachel will provide specific details about how the breakout will work, including logistics, time, etc. 
Some ground rules for sharing with the group include: 


1. Non-advisory participants (NCEO/OESE/OSEP/NCIEA) are generally observers only. However, 
they may be called upon by the facilitator to clarify or provide supporting information if 
necessary. 

2. Participants will follow standard expectations for group work. 

o All will be heard 

Be respectful 

Agree to disagree and move on 

Have a parking lot for tangential issues 


Or OO) 


Materials for each breakout: 


e Participants should have all reports/presentations at hand for reference 
e Two note takers per breakout session. Each will use the note taker template. 
e Panel members will receive a link to a Google doc with the 5 topics outlined via email. 


Breakout Session #1 


How Students with Disabilities are Included in Interim Assessments 
(1:10 — 2:10) 


1:10 — 1:15: Welcome the Advisory Panel members. 


Time is very limited and members have already introduced themselves. Facilitator should politely 
move group into discussion quickly. 


“Welcome everyone. We hope you found the presentation and report informative and useful. There is 
a lot for us to discuss and our time is limited so let’s jump into it. As Rachel explained, we have two 
note takers taking notes for us. Advisory panel members should feel free to add any comments to the 
chat. We've also sent panel members a link to a template in Google Docs. If you choose, you can 
also write your thoughts in the Google doc while we are having our discussion.” 


1:15 — 1:25: Participation 


e §=©6After reviewing the How Students with Disabilities are Included in Interim Assessments report 
and hearing their presentation, what are your observations about student participation in interim 
assessments? (Note to facilitator: Delete similar question from subsequent topics if there is not 
time to address all other bullet points.) 

e To what extent are there groups of students excluded in current interim assessments? 

e What are the specific considerations for: 

o Students with disabilities in general? 
o Students with the most significant cognitive disabilities? 
o English learners with disabilities? 
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How does purpose affect considerations for participation? 
What suggestions do you have for recommendations for participation? 


1:25 — 1:35: Accessibility 


After reviewing the How Students with Disabilities are Included in Interim Assessments report 
and hearing their presentation, what are your observations about student accessibility on interim 
assessments? 

To what extent are there limitations to current accessibility policies and decision-making 
processes? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 

o English learners with disabilities? 

How does purpose affect considerations for accessibility? 

What suggestions do you have for recommendations for accessibility? 


1:35 — 1:45: Role of Standards 


After reviewing the How Students with Disabilities are Included in Interim Assessments report 
and hearing their presentation, what are your observations about role of standards on interim 
assessments? 

What are the standards-based issues that need to be considered when using interim 
assessments to measure what students with disabilities know and can do? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 

o English learners with disabilities? 

How does purpose affect considerations for the role of standards? 

What suggestions do you have for recommendations for the role of standards? 


1:45 — 1:55: Technical Issues 


After reviewing the How Students with Disabilities are Included in Interim Assessments report 
and hearing their presentation, what are your observations about technical issues on interim 
assessments? 

What are the technical issues that need to be considered when using interim assessments to 
measure what students with disabilities know and can do? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 

o English learners with disabilities? 

How does purpose affect technical considerations? 

What suggestions do you have for recommendations for technical considerations? 


1:55 — 2:05: Interpretation and Use of Data 


After reviewing the How Students with Disabilities are Included in Interim Assessments report 
and hearing their presentation, what are your observations about the interpretation and use of 
data on interim assessments? 

What needs to be considered for the interpretation and use of data for students with disabilities 
from interim assessment? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 
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o English learners with disabilities? 
e How does purpose affect the interpretation and use of data? 
e What suggestions do you have for recommendations for the interpretation and use of data? 


2:05 — 2:10: Last Thoughts 


e After listening to this discussion, do you have any last thoughts or considerations you would like 
to share that this group has not discussed? 


Breakout Session #2 


OSEP Results Driven Accountability, State Systemic Improvement 
Plans (SSIPs), and Assessment-related State Identified Measurable 
Results (2:50 — 3:50) 


2:50 — 2:55: Welcome back the Advisory Panel members. 


Time is very limited and members have already introduced themselves. Facilitator should politely 
move group into discussion quickly. 


“Welcome back everyone. We hope you found the second presentation and report informative and 
useful. As we saw from the first break-out session there is a lot for us to discuss and our time is lim- 
ited so let’s jump into it. We have our two note takers taking notes for us again. You should also have 
the note-taking template again so you can follow along. Advisory panel members should feel free to 
add any comments to the chat. We’ve also sent panel members a link to a template in Google Docs. 
If you choose, you can also write your thoughts in the Google doc while we are having our discus- 
sion.” 


2:55 — 3:05: Participation 


e After reviewing the OSEP Results Driven Accountability, State Systemic Improvement Plans 
(SSIPs), and Assessment-related State Identified Measurable Results report and hearing the 
presentation, what are your observations about student participation in interim assessments? 

e To what extent are there groups of students excluded in current interim assessments? 

e What are the specific considerations for: 

o Students with disabilities in general? 
o Students with the most significant cognitive disabilities? 
o English learners with disabilities? 
e How does purpose affect considerations for participation? 
e What suggestions do you have for recommendations for participation? 


3:05 — 3:15: Accessibility 


e After reviewing the OSEP Results Driven Accountability, State Systemic Improvement Plans 
(SSIPs), and Assessment-related State Identified Measurable Results report and hearing the 
presentation, what are your observations about accessibility in interim assessments? 

e To what extent are there limitations to current accessibility policies and decision-making 
processes? 

e What are the specific considerations for: 

o Students with disabilities in general? 
o Students with the most significant cognitive disabilities? 
o English learners with disabilities? 
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How does purpose affect considerations for accessibility? 
What suggestions do you have for recommendations for accessibility? 


3:15 — 3:25: Role of Standards 


After reviewing the OSEP Results Driven Accountability, State Systemic Improvement Plans 
(SSIPs), and Assessment-related State Identified Measurable Results report and hearing the 
presentation, what are your observations about the role of standards in interim assessments? 
What are the standards-based issues that need to be considered when using interim 
assessments to measure what students with disabilities know and can do? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 

o English learners with disabilities? 

How does purpose affect considerations for the role of standards? 

What suggestions do you have for recommendations the role of standards? 


3:25 — 3:35: Technical Issues 


After reviewing the OSEP Results Driven Accountability, State Systemic Improvement Plans 
(SSIPs), and Assessment-related State Identified Measurable Results report and hearing the 
presentation, what are your observations about technical issues in interim assessments? 
What are the technical issues that need to be considered when using interim assessments to 
measure what students with disabilities know and can do? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 

o English learners with disabilities? 

How does purpose affect technical considerations? 

What suggestions do you have for recommendations for technical considerations? 


3:35 — 3:45: Interpretation and Use of Data 


After reviewing the OSEP Results Driven Accountability, State Systemic Improvement Plans 
(SSIPs), and Assessment-related State Identified Measurable Results report and hearing the 
presentation, what are your observations about the interpretation and use of data in interim 
assessments? 

What needs to be considered for the interpretation and use of data for students with disabilities 
from interim assessment? 

What are the specific considerations for: 

o Students with disabilities in general? 

o Students with the most significant cognitive disabilities? 

o English learners with disabilities? 

How does purpose affect the interpretation and use of data? 

What suggestions do you have for recommendations for the interpretation and use of data? 


3:45 — 3:50: Last Thoughts 


e After listening to this discussion, do you have any last thoughts or considerations you would like 


to share that this group has not discussed? 
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Appendix D= 
Proceedings Summary 


Welcome and Introductions 


The meeting opened with welcomes and introductions. Sheryl Lazarus, NCEO director, 
David Egnor, OSEP project officer, Christine Pilgrim, MSIP, and Donald Peasley, OESE, 
welcomed all to the virtual meeting. Following a high-level overview of the meeting and its 
purpose, Rachel Quenemoen, consultant, provided a brief history of inclusive assessment. 
Among the points made during the opening were: 


e There is considerable interest in using interim assessments to measure progress of stu- 
dents with disabilities as well as inform instruction, predict performance, and measure 
learning loss. 


e The meeting results potentially have implications for OSEP and OESE accountability 
systems. 


e Guidance is needed on how to enable all students with disabilities, including students 
with the most significant cognitive disabilities and English learners with disabilities, 
to participate in interim assessments in ways that support valid inferences about what 
they know and can do. 


e Guidance could support measurement of what students with disabilities know and can 
do, which can inform state, district, and IEP team decision making, and ultimately 
improve student outcomes. 


A framework was presented to help organize advisor thinking about interim assessments. It 
included five areas: (a) Participation, (b) Accessibility, (c) Role of Standards, (d) Technical 
Issues, and (e) Interpretation and Use of Data.' This framework was not meant to constrain 
thinking or contributions. 


Two presentations provided background information for advisors. After each presenta- 
tion, advisors broke into two groups for facilitated discussions. Note takers compiled 
notes from the breakout sessions so that the advisors could review them overnight in 
preparation for Day 2 of the meeting. Non-advisors in the breakout sessions were to be 
observers only, unless they were asked by the facilitator to clarify or provide supporting 
information. Discussions were summarized in a set of takeaways that addressed current 
concerns or gaps, what should be, and practical considerations for a better system. The 
two presentations are summarized here. 


Presentation 1. Interim Assessment Practices 


Michelle Boyer and Erika Landl from the National Center for the Improvement for 
Educational Assessment (Center for Assessment) provided a presentation on interim 


' Following the panel meeting, “Interpretation and Use of Data” was revised to “Data Use, Interpretation, and 
Reporting” to better reflect the panel members’ discussion of this component of the framework. 


NCEO 31 


assessment practices. They noted that traditionally interim assessment data have been 
used by teachers and students to support classroom learning. LEAs also have used 
aggregated results from these assessments to support administrative policy. More recent- 
ly, emerging from the disrupted schooling that occurred as a result of the pandemic, 
interim assessment vendors have been publishing “learning loss” studies, and states are 
exploring the potential use of interim assessments to better understand the impact of 
COVID. 


Dr. Boyer and Dr. Land! conducted a document analysis of publicly available test vendor 
materials (e.g., marketing materials, accessibility guides, technical manuals and reports) 
for 13 interim assessments (including several suites of assessments) to learn more about 
vendor claims about the use of these assessments with students with disabilities (Boyer 
& Landl, 2021). They found that: 


e Availability of accessibility features and guidance varied across assessments and 
were often incomplete. 


e Documentation often implied that an interim assessment was appropriate for 
students with disabilities. 


e Attention devoted to principles of universal design during test development var- 
ied greatly. 


e No evidence exists in publicly-available materials that any of the vendors con- 
ducted cognitive laboratory-style studies with attention to students with disabili- 
ties. 


e Even when empirical studies had been conducted to establish that an assessment 
was not biased against demographic subgroups, students with disabilities gener- 
ally were not one of the subgroups considered. 


e No evidence was provided that growth measures derived from the interim assess- 
ments would be meaningful for students with disabilities. 


Presentation 2. OSEP Results Driven Accountability, State Systemic Improvement Plans 
(SSIPs), and Assessment-related State Identified Measurable Results (SIMRs) 


Susan Hayes and Kate Nagle provided an overview of OSEP’s accountability framework, 
RDA, which is used to monitor and support states’ implementation of IDEA. States annu- 
ally submit a State Performance Plan/Annual Performance Report (SPP/APR) to report on a 
variety of indicators including assessment (Indicator 3). As part of RDA, states develop State 
Systemic Improvement Plan (SSIPs) (Indicator 17), which are comprehensive, multi-year 
plans designed to improve outcomes for children with disabilities; within this plan, states 
commit to improving a State-Identified Measurable Result (SIMR) focused on student out- 
comes. Many, but not all states, specified SIMRs that use assessment data as the outcome 
measure. Dr. Hayes and Dr. Nagle also shared information about a new 6-year SPP/APR 
cycle (FFY 2020-FFY 2025). 


Sheryl Lazarus then presented a summary of an analysis of how interim assessments are 
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included in the SSIPs of states with assessment-related SIMRs (Lazarus et al., 2021). A few 
states included data from interim assessments in their SIMRs; many more states included in- 
terim assessments in their evaluation plans as a measure of progress toward the SIMR. States 
also reported on data limitations in their SSIPs. Identified data limitations included: different 
districts used different interim assessments making it challenging to aggregate data across 
districts; some districts that were part of the SSIP cohort did not use any interim assessment; 
and data systems in the state lacked capacity to handle interim assessment data. Only seven 
states included alternate assessment data in their SIMRs; all of the states that included the 
alternate assessment used the state summative assessment as their SIMRs. 


Next Steps 


Sheryl Lazarus thanked the advisors for their thoughtful discussions. She said that NCEO 
would summarize the advisors’ comments, and then send it to them to the advisors to review 
and suggest revisions. Based on these, NCEO would prepare a draft report that contained a 
summary of the reviewers’ comments and recommendations. The advisors would then have 
an opportunity to review the complete report draft. After this review NCEO would make final 
revisions and publish the report. 
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